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The problem of optimum reception of binary Gaussian signals is to 
specify, in terms of the received waveform, a scheme for deciding between 
two alternative covanance functions with minimum error probability. 
Although a considerable literature already exists on the problem, an opti- 
mum decision scheme has yet to appear which is both mathematically 
rigorous and convenient for physical application. In the context of a general 
treatment of the problem, this article presents such a solution. The optimum 
decision scheme obtained consists in comparing, with a predetermined 
threshold k, a quadratic form (of function space) in the received waveform 
xU), namely, 

choose r (s,t) if If x(s)h(s,t)x(t) ds dt < k, 

choose ri(s,«) if J J x(s)h(s,t)x(t) ds dt ^ /,-, 

where r (s,t) and r x (s,t) are the covariance functions while h(s,l) is given 
as a solution of the integral equation, 

JJ r (s,u)h(u,v)rj(v,t) dudv = Ti(s,t) - r (s,t). 

This may be regarded as a generalization of the "correlation detection" in 
the case of binary sure signals in noise. 

Section I defines the problem, reviews the literature, and, togetlier with 
certain pertinent remarks, summarizes principal results. A detailed mathe- 
matical treatment follows in Section II and the Appendices. 

I. INTRODUCTION' AM) SUMMARY 

1.1 Definition and Nature of Problem 

The problem of optimum reception of binaiy Gaussian signals arises 
as a mathematical idealization of a common communication problem. 
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Consider a radio communication link containing a random medium. 
The transmitter sends one of two possible signals with known frequency 
rates (a priori probabilities), and the receiver decides which one of the 
two has been transmitted. Even if the transmitted signals are deter- 
ministic, the observable waveforms at the receiver appear to be random 
owing to effects of the random medium and the ever-present thermal 
noise at the receiver. The task of the so-called optimum (or ideal) 
receiver is to decide, upon observation of the received waveform for a 
finite time, which one of the two signals has been transmitted in such a 
way as to minimize the so-called probability of error. Thus, the problem 
of optimum reception amounts to specifying in terms of the received 
waveform such an optimum decision scheme for given a priori proba- 
bilities. 

It is assumed that the values of the received waveforms at arbitrary 
instants of time during the observation interval, say ^ t ^ 1, are 
jointly Gaussian distributed with means zero and a covariance matrix 
which is determined by either one of two known covariance functions, 
depending upon which one of the two signals is transmitted. Then, the 
above problem may be stated as one of testing simple hypotheses as 
follows: Suppose there are two ensembles of real functions of time t, 
^ t ^ 1, which are statistically characterized as being Gaussian dis- 
tributed with identically vanishing mean functions and two distinct 
covariance functions. A sample (function) x(t) is drawn either from 
the first ensemble with probability a (the null hypothesis: H ) or from 
the second with probability 1 — a (the alternative hypothesis: H{). 
Determine a "critical region" A a (a subset of a space of real functions 
ft) for rejecting H (or accepting Hi) if x(t) belongs to A a and accepting 
Ho if x(t) does not, in such a way that the associated error probability, 

P e (A a ) = aP (A a ) + (1 - a)Pi(Q - A.), (1) 

is no greater than P.(A) for an arbitrary A C fi; where P and Pi are 
two Gaussian (probability) measures defined on (measurable) subsets 
of £2 by the two zero mean functions and two covariance functions. 
Thus, the problem of optimum reception amounts to dividing the func- 
tion space into two parts in such a way that the weighted probabilities 
on them specified by (1) are minimum among all possible divisions. 

There are two features worth noting in this formulation. One is the 
lack of uniqueness of the optimum division as a consequence of adopting 
the minimum error probability as the optimality criterion. Namely, it 
is immaterial whether a certain set N (of functions) with both proba- 
bilities zero, i.e., P (A r ) = = Pi(N), should be a part of A a or ft - 
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A a , since it does not contribute to the error probability P e . Thus, those 
sets upon which /' and I\ vanish can effectively be ignored. The other 
feature is a stipulation that the division be specified in terms of the 
general sample (function), namely, the general element o> of the func- 
tion space ft, so that each sample (a received waveform) can be classi- 
fied as a member of A„ or ft — A a . From the probability theoretical 
point of view, these features dictate specification of the division (or the 
decision scheme) to be made in terms of the "almost all sample func- 
tions" (or "almost surely," "with probability one," etc.) proposition. 
While this offers flexibility in one sense, it presents a restriction in 
another. For example, anticipating the forthcoming results, if the 
division of ft is made by means of a certain u> function on ft, this function 
can be arbitrary or even undefined on the sets of w upon which P and 
Pi vanish. Yet, if the function is defined as a certain limit (or, obtained 
by a limit operation), then the sense of convergence must be at least 
"for almost all sample functions," but not "in quadratic mean (in the 
mean)," "in probability," and "in distribution," which are in general 
weaker. 

The problem of optimum reception of binary Gaussian signals may 
be regarded as a generalization of an almost classical problem in com- 
munication theory, namely, optimum detection of binary sure signals 
in Gaussian noise. It is well known that such detection consists in com- 
paring, with a preassigned threshold, the correlation integral of the 
received waveform and a certain function determined by the two signals 
and noise characteristics. More precisely, let [x, , ^ t ^ 1) be a 
Gaussian process whose covariance function is r(s,t), ^ s,t ^ 1, con- 
tinuous and positive-definite, and whose mean function is either m (t) 
or mil/), both continuous, corresponding to the two sure signals. Denote 
the sample function of the process by x(t) and the threshold by c > 0. 
Then Grenander' shows that if the integral equation 

t r(s,t)g(s) ds = n h (l) - m (t) (2) 

J (i 

has a square-integrable solution, the optimum decision scheme under the 
Ncyman-Pearson criterion is the following: 

choose m n (t) if [ x(t)g(t) dt < c, 
Jo 

(3) 

choose mi(t) if [ x(t)g(t) dt ^ c. 
Suppose the two sure signals in the above problem are replaced by two 
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stochastic (Gaussian) signals and the additive noise is included in these 
signals so that the decision between two sure signals becomes now the 
decision between two Gaussian signals. Furthermore, suppose the 
optimality criterion is changed from the Neyman-Pearson's to the 
error-probability minimization. Then, the problem becomes optimum 
reception of Gaussian signals under the minimum error-probability 
criterion. More precisely, let [x t , ±5 t ^ 1| be a Gaussian process 
whose mean function is identically zero and whose covariance function 
is either r (s,t) or ri(s,t), continuous and positive-definite, with the 
accompanying a priori probabilities a and 1 — a respectively. Then 
what are the counterparts of (2) and (3)? That is, under what condi- 
tions can the optimum decision scheme be specified in terms of a corre- 
lation integral involving the sample function, and what is the decision 
scheme itself? 

1.2 Review of Literature 

Despite momentous foundations laid by Grenander in 1950, little 
progress was made toward rigorous solution of the above problem during 
the succeeding decade, due primarily to restrictions of the mathematical 
scope to elementary probability theory. The majority of the work is 
characterized by two features: (*) use (and misuse) of the classical 
method of likelihood ratio and (ii) attempts to specify the decision 
scheme in terms of some integrals involving the sample function. In 
order to use the classical method, however, the continuous (parameter) 
process must first be "represented" by a (finite) sequence of random 
variables. Thus Middleton 2 and Price 3 sample [xt , 2S t ^ 1} to obtain 
the representing sequence x tl , ■ ■ • , Xt n and form their likelihood ratio 
L: 

ln(x tl , ••• ,x tn ) = I^WT 1 ! 4 

exp |i g E [(7? "V - (^"Vlu^.r,,], ^ 

where Rq 00 and Ri n) are two alternative covariance matrices of .<•/, , 
■•• , Xt H given respectively by (R {n) )ij = ro(Ujtj) and (Ri " )y = 
Ti(U ,tj); i, j = 1, • • • , n. Then, as n — > *> and each sampling interval 
becomes infinitesimal, the decision scheme is specified in terms of the 
limits of the exponent and the factor before the exponential in (4), 
provided these limits exist. .Middleton argues on a formal basis that the 
exponent of (4) becomes an integral 
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x t y t dt, 



where the new process [y t , ^ t ^ 1) is given as one of the solutions 
of a pair of certain simultaneous "stochastic integral equations." Price 
also formally argues that the exponent converges to an integral 



/ / .r,[0,(s,O - go(s,t)]xt ds dt, 



where g and g } are given as solutions of a certain pair of ordinary in- 
tegral equations. 

Davis, 4 Bello 5 and Turin, 6 on the other hand, make orthonormal 
expansions of the process and use the Fourier coefficients as the repre- 
senting sequence. However, the formulation of Davis and Bello is based 
upon a ratio of probability density functions of two sequences of Fourier 
coefficients corresponding to two separate orthonormal expansions, 
which is not a likelihood ratio; while the fundamental notion in Turin's 
formulation is "probability density functions of processes," which are 
unbounded functions in general. 

One difficulty common among all the papers is the total absence of 
convergence proofs for series of random variables. As mentioned in 
Section 1.1, the sense of convergence must be "for almost all sample 
functions." Yet, for example, it is not clear on what ground the ex- 
ponent of (4) should converge for almost all sample functions to those 
stochastic integrals, nor is the existence of the integrals themselves 
shown. 

The other common difficulty, of a more fundamental nature, is the 
lack of optimal ity proofs. Considering the process as an ensemble of 
"well behaved" functions of time, it is intuitively plausible that such 
an ensemble should be "adequately" described by the distributions of 
the "infinitely densely" sampled values of the member functions or by 
the distributions of the Fourier coefficients of some orthogonal expan- 
sions in £.. (the space of square-integrable functions). Namely, the 
continuous (parameter) process should somehow be "representable" by 
a sequence (infinite in general) of random variables. However, the 
optimality of the resultant decision scheme should in general be affected 
by selection of the representing sequence. Obviously, there are in- 
numerable ways of sampling the process, resulting in innumerable 
decision schemes. Similarly, there are as many sequences of Fourier 
coefficients to represent the process as orthonormal bases of £ a • Vet, 
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should all the representing sequences eventually yield the decision 
schemes with the same error probability, the minimum? If not, which 
sequences are the best representations? Even if the best sequence is 
chosen, on what grounds will the error probability remain minimum in 
the limit as n — * °° , since, after all, the classical method is valid only 
for a finite n? 

Note that there is no a priori need for the use of either likelihood 
ratios or representations, so long as the proposed decision scheme is 
shown to have the minimum error probability. In fact, Slepian shows 
interesting special examples (of the singular case) where minimality 
of the error probability is explicitly proved. From a different point of 
view, Parzen recently restores Grenander's basic formulation, where 
what is called the Radon-Nikodym derivative plays the role of the 
likelihood ratio in the classical theory, and puts the sampling method 
on a more rigorous basis. 

1.3 Summary of Main Results and Remarks 

Solution of the problem of optimum reception stated in Section 1.1 
rests on the following two fundamental (measure theoretical) facts: 

(a) If Po and Pi are two Gaussian (probability) measures, they 
must be either (i) "equivalent," i.e., P = P 1 , or (ii) "orthogonal" 
(or "singular"), i.e., P _L Pi ■ 

(b) If Po and Pi are equivalent, there exists a certain nonnegative 
random variable /(cu), called the Radon-Nikodym derivative of Pi 
with respect to P , and a set of co points in 12 such that/(co) ^ a/(l — a) 
can be taken as the desired critical region, denoted by A a in Section 1.1. 
On the other hand, if P and Pi are orthogonal, there exists a set H 
of u points in 12 such that Po(H) = and P\{H) = 1, and the critical 
region can be taken to be such a set H. In short, the following set S a 
serves as the critical region: 

[{/(«) ^ a/ (I -«)) if Po = Pi, 

Sa = \ (5) 

[H if Po J. Pi . 

Thus, the problem of determining the critical region now becomes 
the problem of finding such a random variable and a set H. 

Next, through the use of theory of martingales, the following facts 
can be established : 

For almost all sample functions, 
(i) if (and only if) 



then 
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li.n tr [(WV Ri n) - 27 - R (n) (R^)' 1 } < *, (6) 

lim l n (x tl , ■■• , .«'(„) = /(«) under both hypotheses;* (7) 



(ii) if (and only if) (6) is not satisfied, then 

(0 under null hypothesis, 

lim l»(x tl , ■■■ , .rtj = . (8) 

n-.« [oo under alternative hypothesis, 

provided that the sequence [t t ] is dense in the interval 2ai t ^ 1, where 
"tr" stands for "trace" and the likelihood ratio /,, , together with 
R ln) and Ri ln) , is previously defined in (4).f 

Examination of (7) and (8) in conjunction with (5) immediately 
leads to the conclusion that, irrespective of the hypotheses, 

S a = (lim /„(.»-,, , • • • , xj ^ a/(l - a)}. (9) 

n-*« 

Thus, if x(<i), • • ■ , x(tn) are the values of the sample function (the 
received waveform) x(t), ^ t ^ 1, sampled arbitrarily but with the 
restriction that each sampling interval becomes infinitesimal as n — > oo , 
then the optimum decision scheme becomes the following: 

choose r (s,t ) if lim /„ [x (/,),-••, .r ( <„ ) J < a[ ( 1 - a ) , 

s (10) 
choose ry(s,t) if limZ„[.r(«i) s • • • ,.<•(/„)] ^ a/(l - a). 

71-»» 

Furthermore, according to (i), if the given covariance functions r (s,l) 
and ri(s,0 are such that (6) is satisfied by their covariance matrices 
Ro n) and T?/"' obtained through sampling, then, regardless of whether 
r (s,t) or n(8,t) is the true covariance function, the above limit is finite 
for almost all sample functions, and the error probability associated 
with the decision scheme (10) is minimum. According to (it), on the 
other hand, if r (s,t) and /,(*,/) are such that Ro n) and Ri in) do not 
satisfy (6), then for almost all sample functions the limit vanishes 
if r (s,/) is true, while the limit diverges if ri(s,t) is true; and, inde- 
pendent of the given a priori probabilities, the associated error proba- 
bility simply vanishes, resulting in the case of "perfect reception." 

* Recall that the null hypothesis is the hypothesis that r„(s,t) is the true 
covariance function of the process while the alternative is (he hypothesis (hat 
ri(s,t) is the true covariance function. 

f (6) and (7) are also found in Parzen. 8 
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It should be noted, first of all, that the sequence of sampled values is 
not used to represent the continuous process but to obtain the crucial 
random variable / and set H through formation of the likelihood ratio. 
Secondly, under the assumption of the covariance functions being 
continuous, it can be proved that, regardless of the sampling manner, 
the limit of the likelihood ratio satisfies either (7) or (8), thus yielding 
the same error probability, so long as each sampling interval becomes 
infinitesimal as n — > °o. Lastly, negation of condition (6) can be re- 
garded as a necessary and sufficient condition for perfect reception. 

Having obtained the optimum decision scheme (10), the question 
of possible simplification naturally arises next. Examination of the 
form of the likelihood ratio (4) suggests that, if the limits of the ex- 
ponent and the factor before the exponential exist separately, decision 
scheme (10) may be rewritten in terms of these limits. Such an attempt 
already appears in the literature, as mentioned in Section 1.2. However, 
the crucial mathematical consideration hinges upon the condition under 
which such a procedure can be justified. Here, the following condi- 
tion is shown to be necessary and sufficient : 

lim tr [(i2o (B) ) -1 R™ - I] < « 
lim tr [22 (B) (fli^)" 1 - /] < °°. 

n-»oe 

Note that this condition implies (6), as it should, and excludes the 
case of perfect reception. In fact, condition (11) states not only that 
the sum of two traces converge as condition (6) requires, but also that 
the two traces converge individually. In conclusion: If condition (11) 
is satisfied, then there exist a positive constant /3 and a random vari- 
able such that 

& = lim \R ln) (ft 00 ) -1 |, (12) 

n-»oo 

e = lim E £ [(/?o ,n T I - (A 00 )" 1 ]*/**** (13) 

n-oo .=i ;=i 

for almost all sample functions under both hypotheses; and the optimum 
decision scheme (10) is reduced to the following: 



choose r (s,t) if 9(x) < log (l//3)[<*/(l - a)} 2 , 
choose ri(s,t) if 6U) ^ log (l/0)[a/(l - a)] 2 , 



(14) 



* This does not imply that two different decision schemes yield the same de- 
cision for every sample function; rather, a set N of sample functions, for which 
two decisions differ, give no contribution to the error probability, i.e., Pu(N) = 
= Pi(N). 
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where 0(.r) is the value of 8 for the sample function .r(t), which is ob- 
tained by simply replacing x ti and .v tJ in (13) by x(U) and x(tj). 

Although the above decision scheme is certainly a step toward sim- 
plification compared with (10), it is still inconvenient, if not unfeasible, 
for physical application, since it requires limit operations for each 
received waveform. Yet, so long as the likelihood ratio is formed in 
terms of the sampled values, elimination of the limit operation appears 
to be impossible. Recall, however, the problem of optimum detection 
of sure signals in noise mentioned in Section 1.1. There, the likelihood 
ratio is formed in terms of the Fourier coefficients of the so-called 
Karhunen-LoeVe expansion of the process instead, thus resulting in the 
decision scheme specified in terms of an integral in place of an infinite 
series, as shown by (3). Needless to say, in the present problem where 
there are two covariance functions instead of one, additional mathe- 
matical complications should be inevitable. Nevertheless, an optimum 
decision scheme which is essentially comparable to (3) can be obtained, 
as will now be shown. 

Let Xi ^ X 2 ^ ■ • • and ^i(t), fo(t), ■ • ■ be the eigenvalues and the 
orthonormal eigenfunctions associated with the covariance function 
r (s,t), and, similarly, let mi ^ a<2 ^ ■ • • and <pi(t), <p%(t), • • • be those 
associated with ri(s,t). Then, it can be shown that, under the assump- 
tion of r (s,t) and i'i(s,t) being continuous and positive-definite, the 
integrals 

fc = f xMDdt, i = 1, 2, •••, (15) 

exist for almost all sample functions under both hypotheses, and are 
Gaussian distributed with means zero. Furthermore, the covariance 
matrix determining the joint distribution of £i , • • • , £„ is given by either 

(Qo M )u = M.v, or 

(Q> ( "% - «,, = T,HkU k iU kj , H,j = I *(l)*/(0 &, 

depending upon which one of r<>(s,t) and ri(s,t) is the true covariance 
function of the process. 

Thus the likelihood ratio of £i , ■ • • , £„ becomes 

i = i tf-wv 1 1 ex P |1 1 ± mrr - (& (n) n.^] , hd 

which corresponds to (4). It turns out that, under the previous assump- 
tion on the covariance functions, there is a complete parallel between the 
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two formulations, one based upon x tl , • ■ • , x tn and the other based 
upon fr , •••,£„. Thus, for almost all sample functions, 

(»') if (and only if) 

lim tr [(Qo (n) ) _1 Qi (n) - 27 - Q (H) (ft 00 )" 1 ] < «, (18) 

n-*» 

then 

lim 2„(£i, •••, in) = /(w) under both hypotheses; (19) 

n-»oo 

(m) if (and only if) (18) is not satisfied, then 

fO under null hypothesis, 

lim Uh , • ■ ■ , fc) - | . u t . , .. . (20) 

n-oo ^ oo under alternative hypothesis. 

Then, the optimum decision scheme corresponding to ( 10) becomes: 

choose r (s,t) if lim Ufe(as), • • • , £,(«)] < o/(l - a), 

(21) 
choose n(s,0 if lim Z»[&(z), ••■ ,&»(&)] ^ «/(l - a). 

n-»oo 

where &(&),* = 1, ■*• > ^. are the values of the random variables 
£, for the sample function x(t), namely, 



h(x) = [ x(t)M0dt. 



Again, note first the role of {&}, which is not a representing sequence 
of the process but a means for obtaining the crucial random variable / 
and set H by forming the likelihood ratio. Secondly, it can be shown 
that, under the assumption of the two covariance functions being con- 
tinuous and positive-definite, [?<(*) } can be use< ^ in place of fft(0) 
to form { £,-} , but not any orthonormal basis of £ 2 • Lastly, as before, 
negation of (18) can be interpreted as a necessary and sufficient condi- 
tion for perfect reception. Completing the parallel, if (and only if) 

]imtr[(Qo (n) r 1 Qi (n) -/]<«>, 

(22) 
limtr[Q (B) (Qi ( "T' -/]<«>, 

71-»0O 

then there exist /3 and such that 

= lim|Qo (n) (Q 1 ( " ) )" , |, (23) 
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* = lim ft f, [(Qo ( "V - {Q^r'hUj (24) 

for almost all sample functions under both hypotheses; and decision 
scheme (21) is reduced to the following: 

choose r (s,t) if 0(.r) < log (l//§)[«/(l - a )]\ 

(25) 
choose rj(s,t) if d(x) ^ log (l/ft[a/(l - a)f. 

Returning to the original goal of eliminating the limit operation, 
examination of (24) immediately suggests the possibility of rewriting 
6 as a quadratic form in x t . That is, if one defines 

h (n \s,t) = EEW^WW), (26) 

t=l ;=1 

where 

w° = [(Qo (n) )- 1 - (ft w n«, 

then, from (15), 

= lim [ f x s h in) (s,l)x t dsdt, (27) 

and hij " ; i,j = 1, • • • , n, can be given as a solution of the matrix 
equation 

Qo (n) (W n) ) Qi (B) = Qx (B) - Qo (B) , 

or, more directly, h (n> (s,t) can be given as a solution of the integral 
equation 

f [ ro M (s t u)h w (ufi)n ln) Mdudv = n M (s,t) - r M (s,t), (28) 

where 

ro (n) (s,t) = EA,i(s)^(l), r! (n) (s,0 = £ Efl^(«)*/W- (29) 

.-1 ,=1 ;=1 



Then, the following conjecture should be imminent: 

.1 fl 
'0 ^l) 

where A(s,£) is a solution of 



/ / xMs,t)x t dsdt, (30) 
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/ / ro(8,u)h(UjV)ri(v,t) dudv = n(s,0 — r Q (s,t), (31) 

Jo "O 

which are formally the limits of (27) and (28) respectively. The essen- 
tial part of the above conjecture can be shown to be correct. That is, 
if (31) has a solution h(s,t) such that 

[ ( h 2 (s,t) dsdt < oo, then 

( [ xM*,t)xtdsdt = lira E E W'S.-f; 

Jo Jo n-.=o i'=l ;'=1 

for almost all sample functions under both hypotheses, provided that, 
for all i,j = 1, 2, ••■ , 



o« < 1, o« > E' I «>•; I, 



x7 " 5,y 



j=i 



i - E I «/* - 



^ a:, (33) 



a 7 /. 



where 7v is a positive constant independent of i and j. 

Then the optimum decision scheme (25) is immediately reduced to the 

following desired form: 



■ 

(34) 



(•hoose r (s,t) if / / x(s)h(s,t)x(t) ds dt < log -x (-j— — ) , 
Jo Jo /3 \1 — «/ 

choose r,(s,0 if f f x(s)h(s,t)x(t) dsdt ^ log , ( — — ) . 
Jo Jo (3 \1 — «/ 

Difficulty of the proof lies mainly in the fact that, as n increases, the 
coefficients /),/"' themselves vary with n as well as the number of the 
terms of the sum, yet h (n) (s,t) must approach h(s,t) in such a way that 

lim f [ x,h (n) (s,t)x t dsdt = I I x,h(s,t)x t ds dt 

„-oo Jo Jo Jo Jo 

for almost all sample functions under both hypotheses. This accounts 
for need of the auxiliary conditions (33). The first condition is not a 
restriction in physical application since 



T,au = Em< = [ n{t,t)dt 

i=l ;=1 •'O 



is the average energy of the waveform in the interval ^ t :£ 1, which 
can always be normalized to assure a« < 1. Although the remaining two 
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conditions are restrictive, current knowledge of infinite systems of 
equations does not seem to allow their removal. Thus this calls for a 
future investigation of the degree of restriction imposed by them in 
physicial application. 

As anticipated, there is an apparent correspondence between the 
classical case of sure signals in noise mentioned in Section 1.1 and the 
present case of stochastic signals, namely, between (2)-(3) and (31)-(34), 
except for the fact that the constituent functions in the latter case are 
functions of two variables instead of one. As the integral of the decision 
scheme (3) has a simple physical interpretation (the output of a linear 
filter with g(t) as its impulse response), so does the integral in (34). 
Namely, it is the output of a quadratic filter whose impulse response is 
h(s,t). The advantage of this scheme over the others — namely, (10, 
(14), (21) and (25) — is obvious. Given two covariance functions, the 
impulse response of the filter is uniquely determined by the integral 
equation (31) if a solution exists, and decision is made by comparing, 
with a preassigned threshold, the appropriately sampled output of the 
filter with the received waveform as its input, instead of having to per- 
form the limit operation for each received waveform. 

Finally, it should be remarked that the optimum decision scheme 
above differs formally from those previously obtained by others.* A 
further, and more significant, distinction lies in the assured optimality 
of this scheme, inherent in its derivation, while the optimality of the 
others has yet to be proved separately, f 

II. MATHEMATICAL THEORY 

2.1 Gaussian Processes 

Let \.v t , / f T\ be a real Gaussian process with a parameter set 
T = [0,1] and a finite dimensional distribution function /<',,,...,,„ , which 
is determined by given mean function and covariance function where 
/; , • • • , /„ are an arbitrary finite subset of T. It is assumed that the 
mean function is identically zero on T while the covariance function 
is positive-definite and continuous on T X T. In the present problem 
it is desirable to have an explicit representation of the given process 
[xt , t € T\ on a function space. J 

Let 9. be a space of real-valued functions of / £ T. Let .r f (o>) be the 

* Although their work is briefly reviewed in Section 1.2, their derision .schemes 
are not stated explicitly in this paper. 

t This excludes Par/en's* case where the decision scheme is essentially (10). 

% The next paragraph follows close] v Example 2.3 in Supplement, Doob, 9 pp. 
609-610. 
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oj function with the value f(s) if w is the function J-( • ), so that x,(«) = 
f(s). If the * function a> has values f(£ x ), ■ • • , £(*») at ^ , • ■ • , f„ , 
the condition 

f (/i) ^ pi , • • • , r(*») ^ p» 

defines an to set, which is denoted by 

[x tt (o) £p,, i= 1, ••-,»} (35) 

where pi , • • ■ , p B are arbitrary real numbers. Next, let £F be the class 
of all co sets obtained in this way for arbitrary n, k , • • • , t n , and let 
(B T be the Borel field generated by fF, and lastly let P be a probability 
measure defined on the sets of (Br whose value is given by 

P{x ti (<a) ^ Pi , i = 1, • ■ • , n\ = F h ,..., ln (p, , • ■ • , p n ). (36) 

Then, {x t (co),t € T\ is a representation of the given process \x t , t 6 7 1 } 
on the function space fi, and (fl, <B T , P) is the explicit probability meas- 
ure space for the representation. 

(Remark) By virtue of the choice of representation space, the general 
elements of the space ft coincide with the general sample functions of 
the process [x t (u), t € T\. Thus, the phrases, "almost everywhere (or 
almost surely)" and "for almost all sample functions," have the same 
meaning. 

The assumption of continuous covariance function has the following 
significant consequences : 

(i) \x t (w), t € T) has an equivalent (with respect to P) separable 
and measurable process on the same o> space, f Hence, so long as the 
almost-everywhere valid properties of a given process are of interest, 
as in the case of this paper, the given process may as well be taken 
to be separable and measurable. Therefore, the Gaussian process 
(X((w), t € T\ is henceforth assumed to be separable and measurable. 

(ii) \xt(co), t £ T\ is sample (Lebesgue) square-integrable on T 
almost everywhere with respect to P.% 
This immediately implies that a Lebesgue integral 

* Symbolic distinction between the given process and its representation on the 
function space is made by explicitly writing the argument w for the latter. 

f Note continuity of the covariance function of a process is equivalent to con- 
tinuity in quadratic mean of the process (Lodve, 10 p. 470), and hence it implies 
continuity in probability of the process. Then, according to Theorem 2.6 in Doob, 9 
pp. 61-62, there exists an equivalent separable and measurable process on the 
same space. 

\ See Loeve, 10 pp. 520-521. 
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{(«) = [ x t (a)f(t) dt 

J T 

exists almost everywhere, in which yp{t) is any continuous function on T. 
Furthermore, since the sample Lebesgue integral of a process coincides 
almost everywhere with the Riemann integral in quadratic mean cri- 
terion,* and also the Riemann integral in quadratic mean criterion of a 
Gaussian process is a Gaussian (random) variable, f £(w) is a Gaussian 
variable. 

2.2 Formulation of Problem 

Let Fo ; ti,---,t n and F\- *,,.-.,/„ be two alternative Gaussian finite di- 
mensional distribution functions of a real separable and measurable 
process [x t (u), t 6 T\, whose mean functions are identically zero and 
whose covariance functions, denoted respectively by n{s,t) and n{s,t), 
are positive-definite and continuous on T X T. Let Po and Pi be the 
Gaussian probability measures defined respectively by F^ ltl ,.-.,*„ and 
F 1; t lt —,t„ on the Borel field (Br of subsets of 12 as defined previously. 
It is well known that P and Pi are either equivalent, Po = Pi , or 
orthogonal, P J_ Pi 4 

Define a set function P e by 

P C (A) = aPo(A) + (1 - a)Pi(Q - A), A 6 <B r , (37) 

where a is a constant, < a < l.§ Let A a 6 (Br be such a set that 
Pe(Aa) ^ P,(A) for all A € (Br. (38) 

Then, the problem of interest is to specify such a set A a in terms of 
x t {(a). || 

Now, if Po = Pi , let f(co) be a Radon-Nikodym derivative of Pi 
with respect to P ; while, if P J. Pi , let H € (B r be a set such that 
Po(H) = and Pi(H) = 1. Then, it can be shown that the following 



* Henceforth, the "sample Lebesgue integral of a process" will simply be called 
the "integral of a process," unless otherwise specified. A definition of Riemann 
integral in quadratic mean criterion is in Loeve, 10 pp. 471-474. 

f See Loeve, 10 p. 485. 

t See Hajek. 11 - 12 

§ P e is the so-called error probability. Although ^ P e ^ 1 for all A 6 ffi r , 
Pe is not a probability measure, and its full meaning is given in Section 1.1. 

|| Equivalence between this problem and that of "optimum reception of binary 
Gaussian processes" is discussed in detail in Section I. 
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set S tt satisfies condition (38): 

if P Q = P 1} S a = {/(«) ^ a/(l - a)}, 

if Po J- Pi , S a = //. 

Thus the above stated problem is reduced to that of finding 

(i), if P = Pi , a function of x t (<a) equal to/(o>) almost everywhere 

with respect to Po and Pi , and 

(u)i if Po _L Pi , some such set // expressible in terms of x t (u). 

2.3 Solutions — / 

2.3.1 General Solution 

Let {t a } be a sequence of points in T = [0,1], which is dense in 7\ 
Let (B„ be a Borel field generated by a class of a> sets of the form 

[x Tt (u) ^ Pi , i = 1, ••■ ,n}, (40) 

and let (B M be the minimal Borel field containing U (B„ . Obviously, 

n=l 

(Bi C (B 2 C • • • C (B w C (B T . (41) 

Then, since {.t«(oj), f € jT) is a separable process, continuous in proba- 
bility (with respect to P and Pi), and the sequence jr*} is dense in T, 
it follows that, for an arbitrary set A G (Br , there exists a set A' € (B„ 
such that 

Po(AAA') = = Pi(AAA'). (42) t 

Now, through the use of the covariance functions r (s,t) and ri(s,t) 
and the fact that the mean functions are identically zero, the density 
functions po and pi of the random variables x Ti (w), i = 1, ■•• , n, 
corresponding to P and Pi respectively, are obtained as follows: 

P«(^ ,•••,"») = (**) I /Cm | 

r i - » l (43) 

X exp i-t g g [(B- W )1tfw). « - 0, 1, 

where the r t , t = 1, • • • , n, are a finite subset of { n-} , and P,„ " , m = 0, 
1, are n X n symmetric, positive-definite matrices defined by 

(Rj n) )a = r m {n , tj); m = 0, 1; i, j = 1, • ■ ■ , n. (44) 

* See Appendix A. The first assertion of (39) follows from Corollary 1 in this 
appendix, while the second assertion is self-evident. 

t .See Doob, 9 pp. 51-55; in particular, Theorem 2.2 (i). 
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Then define a random variable l„(u) by 

«»(«)= — — — - - | «0 («! ) | 

poUv.tw), ••-, .r T „(w)] 

(45) 

X eX P {lZ;[;[(/?o , '' , r 1 - (fll (W TVr>)*r,(co)J. 

Note that 

/„(«» ^ for all «. (4fi) 

Furthermore, since R m {n \ m = 0, 1, are positive-definite, pi = when- 
ever p = and vice versa. Then, it can be shown that the processes 
{/ n (w), n ^ lj and [l/l„(u), n ^ 1| are martingales with respect to 
Po and Pi respectively. 

(i) p = P, : Let 2?o{/(w)|<B„}, n = 1, 2, • • • , be a conditional 
expectation of /(«), given (B„ , with respect to P • Namely, 

J #o{/(«) | (?„} dl\ = f f(co) dl\ for any A € (B„ . 

Then, 

Z n (a>) = tfo{/(w)| (B„|, a.e. (P„),t (47) 

and, from (41) J 

lim £„{/(") |® fl } = E {f(u)\(Bj, a.e. (P ). (48) 

Yet, from the definition of /*,'„!/( a- ) | (Bj and (42), 

E [f(a,)\(&J = f(u>), a.e. (P c ). (49) 

Hence, 

lim/„(co) = /(w), a.e. (P ). (50) 

n-»so 

Since P = Pi , the above implies 

lim Z„(u) = /(w), a.e. (Pi). (51) 

n-»oa 

Thus, the desired function, which is equal to /(w), a.e. (P , Pi), is 

* See Doob, 9 pp. 91-93. 

t "a.e. (Pm)," m = 0, 1, is used as a shorthand notation of "almost everywhere 
with respect to P m ■" Similarly, "a.e. (P , Pi)" will be used to denote "almost 
everywhere with respect to both Po and Pi ." 

JSee Doob, 9 p. 331. 
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l n (u), which is defined by 

iju) = lim *.(«). (52) 

n-»oo 

(tt) Po ± Pi : From (46), lim /„(«) < », a.e. (P ).* In fact, it can 
be shown that 

lim/„(«) = 0, a.e. (P ).t (53) 

By using the same argument, it follows that 

lim [l/Z»(o>)] = 0, a.e. (Pi). (54) 

n-»oo 

Hence, for an arbitrary constant c > 0, 

P {limJ«(w) £ c} - 0, Pi{limZ„(«) £ c} = 1. 

n-»oo n-»oo 

Thus, the desired set //, with P (H) = and Pi(#) = 1, is 

H = {limZ„(o>) ^ a/{\ - a)\. (55) 

n-»oo 

In summary, upon combination of (52) and (55) in conjunction with 
(39), the desired set S a is 

S a = (lim i.(«) ^ a/(l -a)}, (56) 

i irrespective of whether P = Pi or Po _L Pi . 

2.3.2 Special Solutions and Summary 

Under certain restrictive conditions, the set S a can be specified in 
terms of well defined functions of x t (u>). It is the purpose of this sub- 
section to obtain such specifications as well as the accompanying con- 
ditions in terms of the given covariance functions ro(s,t) and ri(s,t). 

(i) If 7*0 = Pi , it has already been shown that 

S a = {/.(«) ^ a/(l ~ a)\. 

Thus, it is of interest to obtain a condition for P = Pi . t 
Define 

,„(») = [/.(«) - 1] log /„(«), n = 1, 2, • • • . (57) 



* See Doob, 9 p. 319; Theorem 4.1 (i). 
t See Doob, 9 pp. 345-34(5. 

j Such conditions are already available (e.g., Parzen, 8 Shepp 13 ). For more de- 
tail, see Yaglom. 14 
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Then since (p — 1) log p, p > 0, is a real, continuous and convex 
function of p and E \\(l„(ta) - 1) log /„(w)|} < *>, n = 1, 2, • • • ; 
{7j n (w), n ^ 11 constitute a semi-martingale (with respect to P ). 
Hence, E {rj n (u})}, n = 1, 2, ••■ , forms a monotone nondecreasing 
sequence 

-EoM")} ^ £•!*(«)} ^ ••• , (58)t 

which must either converge or diverge. Then, according to (53), 

if P X Pi , then 

lim-EoM")} - °°. (59) 

n-*oo 

Hence, since P and Pi can be either equivalent or orthogonal, it follows 
that 

Po s P, , if 

lim E [ Vn (o>)} < co. (60) 

n-»oo 

It can be shown that the converse of (59) is also true, J i.e., 

if limtfoM")} = °°i thenPoXPi. (01) 

»-»oo 

This implies that the condition of (60) is also necessary. Thus, through 
substitution of (45) into (57) and application of (43) for expectation 
calculation., § 

P = P x , if and only if 

lim tr [(W )" #/'" " 2/ + R in> (lO -1 ] < •■ (62)|| 

where flo (n) and #i (B) are defined in terms of r (s,t) and n(«,0 by (44). 
(tt) Examination of (45), (50) and (51) indicates that, in addition to 
condition (62), if 

lim | fl (B) (/2i (n) )~ I I = ft < < *, (03) 

n-»oo 

then 

* See Doob, 9 pp. 295-29(5, Theorem 1.1 (Hi). "E a " denotes expectation with 
respect to P , namely, an integration over Q with respect to P . 

t See Doob, 9 p. 324, Theorem 4.1s. 

X See Hajek; 12 in particular, Lemma 2.1. 

§ For this calculation, use the following equality: E u \rj„(f)\ = A\|log M")) — 
£o|log /„(«)), n = 1, 2, •■• . ... 

II "tr" denotes "trace," and / is the n X n identity matrix. 
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lim 2 EK^ 00 )" 1 - (i? 1 (n) )- , ] l7 .r T ,( w ).r r; .(c) < », 

«-« i=i ,=i ( 64 ) 

a.e. (P ,Pi). 

Thus, by defining 0(a>) as the above limit, i.e., 

o( U ) - lim e i: [(/?o (n ')- 1 - (ft^ni^r^-)*,^-), (65) 

the set »S'„ f (B r can be specified as follows: 

S a = [0(a) ^ log (l/(3)(a/(l - a)) 2 \. (66) 

It will now be shown that two conditions (62) and (63), required 
for the above specification of >S'„ , are equivalent to the following pair 
of conditions: 

limtr[(/2o (B) ) _1 «i (n) -/] < oo, 

R-*00 

and (67) 

lim tr [R (n) (ft,"") -1 -/]<«,. 



Define 



Thus, 



$•»(«) = -log/„(a>), 

(68) 
£„'(«) = l n (o>) log/„(co) n = 1,2, ••• . 



i?»(«) = £„'(«) + f n (co), n = 1, 2, • • • . (69) 



Again, just as in the case of j7„(w), both {("»(&>), n ^ 1J and {f„'(w), 
n ^ 1} are semi-martingales with respect to P , and 

Eo\M*>)\ ^ Eo\Mo>)\ £ ■■• , 

(70) 

^o{fi'(«)} ^^oiri'(«)l ^ ■■■■ 

Furthermore, from (53), 

if PtJ.Pi, then lim^ {f«(o>)} = «. (71) 

n-»oo 

However, from (69) and (70), divergence of i?o{ £»(&>)} implies that of 
E a [rj„{u)\. Hence, according to (61), the converse of (71) holds. Then, 
again from (70) and the equivalence-or-orthogonality dichotomy of 
P and Pi , ' 
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Po m Pj if and only if lim # {f»(«)} < ». (72) 

n-»oo 

Thus, upon substitution of (45) into (68) and application of (43) 
for expectation calculation, an alternative necessary and sufficient 
condition for P = P, is obtained as follows: 

lim {log | (PVT' Ri (n) | + tr [Ro (n> (R^r 1 - /]} < •. (73) 

n-»oo 

Now, under the condition (03), the above condition implies that 

lim tr [/?„"" (/?/"')"' - J] < *. (74) 

II -.00 

Then, upon combination of conditions (62) and (74), condition (07) 
immediately follows. 

The result of this section may be summarized as follows: 

(t) In general, 

S a = |lim/ n (co) > o/(l - a)}, 

n-»oo 

where /,,(a>) is defined by (45). 

(u) If P — Pi , which is true if and only if 

lim tr [(ffo ( "T' Ri M - 2/ + R in) (^ 1 < " , )"'j < «>, 

then lim /„(w) = /(«), a.e. (P n , P,); thus by defining /^(co) = lim /„(a>), 

S a = [lju) ^ a/0 ~ «)!• 



(m) if 



lim tr [(flo 00 )" 1 ft 00 - /J < », 

lim tr [^o"" («i (n, )~ 1 - 7] < oc, 



then 



S« = |0(cu) ^ log (l/|8)(a/(l - a)) 2 } 
where 0(a)) and are defined by ((i5) and (63) respectively. 

2.4 Solutions — II 

2.4.1 General and Special Solutions 

Let Xi ^ X^ ^ • ■ • and ^i(/>, fo(t), • • ■ be the eigenvalues and the 
corresponding orthonormal eigenfunctions associated with the covari- 
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ance function r (s,t).* Similarly, let mi ^ M2 ^ •• • and <pi(t), tp*(t) } ■ ■ ■ 
be such eigenvalues and eigenfunctions associated with ri(s,t). Then, 
according to the discussion in 2.1 («), continuity on T of each fc{t) 
implies that the integrals 

fc(w) = ( x t (u)Mt) dt, i = 1, 2, • ■ •, (75) 

exist a.e. (Po , Pi), and are Gaussian random variables. In fact, it 
can be shown that the density functions p and p x of &(«), • • • , £»(«) 
corresponding to P and P x are given byt 

„ / \ /O \-("/2) I n (n) I-, 1 

p»("i » '•■ , v n ) = {2ir) I (4m I 

r l » » i (76) 

exp |-,EZ [(Qm M r l h wj , m - 0, 1, 



2 ,=1 ,=1 

where Q m (n) , m = 0, 1, are n X n symmetric and positive-definite 
matrices denned by 

00 

(Qo <B) )« = Mi> . (Qi (n) )o- = Z MM** , (77) 

A- = l 

where 

un = [ <pi{t)yPi{t) dt. (78) 

Let (B„ be a Borel field generated by a class of o> sets of the form 

{&(») ^ P*,» = 1, ••• ,«}, (79) 

00 
and let <6_ be the minimal Borel field containing U (&„ . Obviously, 

n=l 

(Bi C 4 C • • • C^Cffir. (80) 

It can be shown that, for an arbitrary A 6 ($> T , there exists some A € & M 
such that 

Po(AAA) = 0. (81) % 

Now define a random variable L(w) by 



* More precise definitions of these eigenvalues and eigenfunctions are given 
in Appendix B. 
t See Appendix C. 
\ See Appendix D. 
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fr[&(»)> •••.£»(«)] 
= iQo^CQi^r'Pexp^EEKQo^r 1 (82) 

- (Qi (n) )~W»)fc(«)} 1 

where (76) is substituted for the second equality. Again, note that 
l„(o)) is nonnegative for all n and also the fact that pi = whenever 
■po = and vice versa since Q m ln) , m = 0, 1, are positive-definite. Thus, 
again the processes {!»(«), n ^ 1} and {l//„(w), w ^ 1} are martingales 
with respect to P and P t respectively. 

By following step-by-step the same procedure as the one in the 
preceding section,* the following results are obtained : 

(i) In general, 

S a = {limL(a>) ^ a/(l - a)}. (83) 

(ii) If P Q = Pi , which is true if and only if 

lim tr [(Qo M )~ > Qi (n) - 2/ + Q <B) (Qi (n) ) -1 ] < », (84) 

n-»oo 

then 

lim L(«) = /(«), a.e. (P , Pi); (85) 

n-»oo 

thus by defining 

tju) = lim L(w), (80) 



S„ = (U«) ^ «/(i - «)}■ ( 87 



(in) If 



limtr[(Qo < " ) )" 1 Qi <n) -/]<», 

(88) 
limtr[Qo (n) (Qi (n, ) _1 - 7] < *>, 

then there exists a constant p, < $ < ■» , such that 

* In effect, it amounts to replacing (B„ and /„(«), n = 1, 2, ■ • ■ », by (B and 
Z„(a>) respectively. 
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lim|Qo ,n> (Q 1 <n) )-'| = ft (89) 

and, from (85) and (82), it follows that 

»-«. »tf £l (90) 

a.e. fPn,Px); 

thus, by denning 0(o>) as the above limit, 

S. = {*(«) ^ log (l//3)(«/(l - a)) 2 ). (91) 

2.4.2 Integral Expression for 6 (to) 

For the purpose of physical application, it is desirable to express the 
random variable 0(w) as a simpler function of x t (a)), in particular, with- 
out involving limit operation. Examination of the definition of 0(w), 
i.e., 

0(«) = lim E £ [(Qo'"') -1 - (Qi C,0 rW»)&(»), (92) 
indicates that 0(u>) might be expressible as a quadratic form in Xt(u), 



i.e.. 



[ [ x t (a)h(s,t)x t (a) els dt 



if such a square-integrable function h(s,t), (s,t) 6 7 1 X T, exists and 
can be determined uniquely. It is the purpose of this subsection to make 
the above statement more definite and precise. 
Define an n X n symmetric matrix H " by 

//<«> = (Qo <«))-> _ (&<«>)-'. 
Then, 

or, through (77), the equation for the i-jth. element becomes 

2 \i(H {n) U(Qi (n % = (Qi^a - \i8 ii: i, j = 1, • • • , n. 

t=i 

In other words, every ith row of H (n) satisfies the following system of 
equations : 

£ a jk hik (n> = bj(i) j = 1, ■■• , n, 



Note that the solution is unique, since the matrix (o,-,0 is positive-definite. 
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whore 

a, j = 7^ VkUkiUkj , 

« i,i = 1,2, .. , (93)* 

6j(0 = (aij/\i) - bn , 
or its standard form 

W" = Z c,*W° + 5,(t), ; = 1, • • • , n, (94) 

where 

Cn = Stj — «ii • 

Now, for each i = 1,2, • • • , consider the following infinite system of 
equations : 

ha = E c ]k h ik + b^i), j - 1, 2, • • • . (95) 

According to the theory of infinite systems of equations,! if (95) has a 
solution (/(.i , h it , ■ ■ • ) for each j = 1, 2, • • • , such that 

i: £w < », on) 

i=\ 7-1 

then (/t,t , ha , • • • ) is unique and 

hi, = lim h,/ n) , j - 1, 2, • • • , (97) 

n-*oc 

for each i = 1, 2, • • • , where (ha"\ • • • , fei» (B) ), * - 1, ■ • ■ , n, ia 

the solution of (94); provided that (95) satisfies the following condi- 
tions: for each i = 1, 2, • • • , 

X>,/|<1, (98) 

and there exists a constant Ki > 0, independent of j, such that 

| bj(i)\ ^ Ki(l - g | c* |), j = 1, 2, • ■ • . (99) 

On the other hand, if (ha » ^« » - • ■ ) is a solution of (95) for each 
i = 1, 2, • • • , satisfying (96), then the following integral equation 

nr (s,u)h(u,v)ri(v,t) du dv = i\(s,t) - r (s,t) (100) 



* Note that (Qi<">),-,- = a<y ; i, j = 1, ■ ■ ■ , n. 
t See Kantorovich and Krylov, 16 pp. 20-33. 
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has a square-integrable solution h(s,t), 

[ f h 2 (s,t)dsdt < oo, (101) 

such that 

h(s,t) = £} IM^)*^), in the mean.* (102) 

»=i y-i 

Conversely, if /l(s,0 is a square-integrable solution of (100), then (95) 
has a unique solution (hii , h i2 , ■ • • ) for each i = 1,2, ■■• , satisfying 

00 00 

Z) 2 V < °° > sucn tnat 

i 1 ;-l 

£y - [ [ Ms)ti(s,t)+j(t) ds dt.\ (103) 

J fj t 

Now, extend the definition of /f,/"', i = 1, 2, • • • , n,t by adding 

W n) = 0; », i = n + 1, n + 2, • • • . (104) 

Then, (90) and (92) can be rewritten as 



and 



lim £ £ W n) &(«)&(«) < », a.e. (P, , PO, (105) 
«-*oo t=i y=i 



d(») = lim 2 D W n) &(«)&(«)• (106) 

n=oo t=l ;=1 



According to the theory of coordinate and projective limits in sequence 
spaces, § (97) and (105) imply that 

*(«) =EE *<*«<«)&<«). a.e. (P. , P,), (107) 

.=i y=i 

since 

Z Z tf («)€/(•) < -, a.e. (P, , Pj). (108) 

1=1 y=i 
On the other hand, from (102) and square-integrability, a.e. (P , Pi), 

* See Appendix E.l. 
t See Appendix E.2. 

t Namely, (ha , ■■■ , h in ) is the solution of (94) for each i = 1, • • • , n. 
§>See Cooke, 16 pp. 282-289; in particular, Theorem (10.3, II), extended to the 
case of double sequences. 
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of{xM,t € T), 



J mj m 



x a (o})h(s,t)x t (co) dsdt = 2 EM<(«)6'(«). 

« »-i (109) 

a.e. (P ,Pi). 

Hence, 

0(c) = f f ^(w)A(s,0aj«(w) <fc<ft, a.e. (P ,Pi). (HO) 

2.4.3 Discussion and Summary 

Recall that, in order to specify the set S a 6 (Br for given a as (91), 
it is sufficient to assume (88), which assures existence of 6 (a>) and p 
defined by (89) and (92) respectively. Moreover, in order to express 
6 (to) as (110), it requires the additional assumptions that (i) the 
integral equation (100) have a square-integrable solution and (ii) 
the conditions (98) and (99) be satisfied. 

It can be shown, however, under the assumptions (i) and the fol- 
lowing : 

an < 1, i=l,2, •■■, (111) 

the conditions (ii) and (88) can be replaced by the following: 



«,»> Z'lotfl. i= 1,2, ••• , (112) 



and that there exists a constant K > 0, independent of i, j = 1, 2, 
• • • , such that 

KowA*) - «« I ^ *(«« - h I «i* I), (H3) 

where a,-/ is defined by (93).t It is quite possible that, once the condi- 
tion (i) is assumed, the conditions (111), (112) and (113) may be 
superfluous. That is to say, in some special cases, if the integral equa- 
tion (100) admits a square-integrable solution h(s,t) it may be pos- 
sible to prove directly that 

h(s,t) = lim £ f) WW«)*y(0. (H4) 

n-»oo i=I j=l 

in the mean, which immediately implies (97) and (105), thus establish- 

* The prime on the summation sign symbolizes omission of the term j = i. 
t See Appendix D. 
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ing (107) and leading to (110). However, in the general case, establish- 
ment of (114) does not seem possible, nor does finding a sufficient 
condition for (114), without making the resultant condition excessively 
implicit and complex. 

2.5 Summary 
If 

/ / r (s,u)h(u,v)ri(v,t) du dv = ri(s,t) — r (s,t) 

J T J T 

has a solution h(s,t), 

f I h 2 (s,t) dsdt < oo, 
then the set S a 6 (B r , given a (0 < a < 1), can be specified as 

S a = {// x,(a>)h(s,t)x t (o,) dsdt ^ logj(r-^-)\, (115) 

where 

$ = lim|Q (n) (Qi^T'l, 

«-*oo 

and 

(Qo M )ij = K8» , (Qi (n) ).i = an ; *, j = 1, • • • , n, 
and 

aij = £ UkUuUkj ; i,j = 1, 2, ■ • •, 



w.7 = / <Pi(t)fj(t) dt; 

J J! 



where Ai ^ X 2 ^ ■ • • ; ih(<)i fo(0, • • • , and mi ^ M2 ^ • ■ ■ ; M0, 
(p 2 (t), •■• , are the eigenvalues and the corresponding orthonormal 
eigenfunctions associated with the given co variance functions r (s,t) 
and n(s,t), which are positive-definite and continuous on T X T; 
provided that 

(l)a« < l,t= 1,2, ■•• , 

00 

(2) an > E' I «-> I. «' = 1, 2, • • • , 

(3) the following is bounded uniformly in i, j = 1,2, • • • : 



RECEPTION' OF BINARY GAUSSIAN SIGNALS 2795 



fly . 

x7 8ij 



< K. 



1 - Z I «i* - a* I 
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APPENDIX A 

Theorem on Optimality 

Let P and Pi be probability measures defined on a Borel field (B 
of subsets of an abstract space fi. Through the use of Lebesgue decom- 
position theorem and Radon-Nikodym theorem:* for a nonempty set 
H 6 03 with Po(H) = 0, there exists a nonnegative function f(a>) in- 
tegrable over Q with respect to P such that 

Pi(A) = { f(u)dP + Pi(A fl H) (116) 

for an arbitrary A £ 03. t 

Theorem: For an arbitrary constant k > 0, define a set S 6 ® by 

S = {/(«)' ^ k] \JH. (117) 

7%era, 

kPo(S) + Pj(S c ) - fePo(A) - P,(A C ) g (118) 

for an arbitrary set A 6 03 wfare S e and A c are /fee complements of S and 
A tw#A respect to £2. 

Proq/V 

Put p = kP (S) + Pi(iS c ) - fcPo(A) - Pi(A c ). By adding and 
subtracting kP Q (S fl A) and P,(»S C fl A c ), 

* See LoeVe, 10 pp. 130-132. 

t This paragraph closely parallels Grenander, 1 pp. 209-210. 
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p = k[p (s) - p (s n a)] + p^s") - p 1 (s c n a c ) 

- fc[P (A) - P (S n A)] - P,(A C ) + Pi(S c PI A c ) 

(119) 

= kp (s n a c ) - p^s n a c ) + Ptis* n a) 

- kP (S e n A). 

From (116) and (117), with k > and P (#) = 0, 

Pris n a c ) - p^s' n a) 



= f f(u)dP» - f /(co)dPo 



(120) 

+ Px(5 n a c n h) - Pds" n a n //) 
^ &p„(s n a c ) - kp (s c n a), 

since 

P 1 (S C C\Af\H) 

=Pi({Ku>) < k\ n h c n a n h) ^ Pi(# c n#) = o. 

By substituting (120) into (119), 

which proves (118). (Q.E.D. ) 

Corollary 1. Suppose Po = Pi, and let k = [a/(l — a)], < a < 1. 

T/iew, a se£ $ a defined by 

S a = {/(«) ^ a/(l - a)} (121) 

/wis the properly expressed by (118), i.e., 

aPo(S a ) + (1 - a)Pi(5 a c ) ^ aPo(A) + (1 - a)P x (A c ) (122) 

for an arbitrary A G (B. 

Proof: 

Note that P ■ Pi implies P X (H) = 0. Hence, in (118), 

kP (S) + PtiS') = [a/(l - a)]P (S a UH) + PiCS. U IT) 

= [a/(l - a)]P (5.) + Pi(-S«). 

Thus, substitution of the above into ( 1 18) and multiplication by 1 — a 
proves (122). 

Corollary 2. Take ft to be R n , an n-dimensional Euclidean space, and (B 
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to be Borel field of right semi-closed, semi-infinite intervals in R n , denoted 
by 5 n . Let p m (xi , ••• »z»), m = 0, 1 and (x x , • • • ,Xn) € R n , be Baire 
density junctions corresponding to P m , m = 0, 1; i.e., 

P m {xi £ p i} i = 1, •••yd) 

rPi rPn (123) 

= / ■••/ dxx • • ■ dx n p m (xi, • ■ -,x n ). 

J— oo •'—oo 

Suppose pi(xi , ■ • ■ ,x„) = whenever p (xi , • • • , x n ) = 0. Then 
S a ,n defined by 



[po(xi, • • -,x n ) \ — a 
has the property expressed by (122). 

Proof: 
Note that P = Pi , thus Pi(#) = 0. Then, from (116), 

/(*i , • • • , *n) = P f 1 '' "' X *\ , a.e. (P , Pi) 
Po(xi, •",**) 

Hence, apply Corollary 1 . 

APPENDIX B 

Preliminaries on Integral Operators] 

Let L be an integral operator with a real, symmetric, continuous and 
positive-definite kernel r(s,t) defined on the rectangle T X T where 
T is the closed interval [0,1]. That is, 

Lf(t) m f r(s,t)f(s) ds , (124) 

where /(0 is an arbitrary real-valued function in the space of square- 
integrable functions on T, which is symbolically denoted by £ 2 (0,1), 
or simply by £ 2 • 

Then, according to the theory of linear operators, all the eigenvalues 
of L are positive, of finite multiplicity, and finite or denumerably in- 
finite in number. Thus, counting each eigenvalue as many times as its 
multiplicity, we can construct an ordered sequence of eigenvalues, 

* This replaces the Neyman-Pearson theorem in the classical theory of testing 
simple hypotheses when the criterion changes from the Neyman-Pearson 's to 
the minimum error probability. See Cramer, 18 pp. 529-530. 

t See Riesz-Nagy, 17 pp. 227-246. 
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Xi ^ X 2 ^ ••• , (125) 

and the corresponding sequence of orthonormal eigenfunctions (using 
the Gram-Schmidt ortho normalization process if necessary) , 

M),Ut), ••• • (126) 

Then, according to Mercer's theorem, 

r(s,t) = t^«)WO, (127) 

where the series converges uniformly on T. Consequently, \J/i(t) is 
continuous on T for all i, and 

£)Ai = [ r(t,t) dt< * , (128) 

namely, the sum of all eigenvalues is finite. 

Furthermore, because of the positive definiteness of the kernel r(s,t), 
the set of the eigenfunctions {^(0} forms an orthonormal basis of £ 2 . 
Let {ipi(t)} be another orthonormal basis of £ 2 . Then, 

iMO = ]C u <i<Pi(0, in the mean, (129) 

i=l 

where 

un = f *.■(*)*,(*) dt, (130) 

J f 

which satisfies the following orthogonality conditions: 
2 UikUjt = / (pi(t)(pj(t) dt = 8,j , 

2 u kiUkj = / ^i{t)yp,{t) dt = 8 (j . 

1—1 J T 



(131) 



t-i 



APPENDIX C 



Density Functions of &(«), * = 1, ■ • • > » 

It has been established in Section 2.4.1 that the random variables 
defined by (75), i.e., 

&(«) = / &«(«) *«<*) *, * - 1. 2, • • •, (132) 
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are Gaussian variables with respect to P and Pi , where 

E m {x t (w) } = 0, t e T, m = 0, 1, and f x t 2 (a) dt < «>, 

a.e. (Po,Pi) . 

C.l With respect to Po 

Through repeated use of Fubini's theorem, 

EoltM \ = { Eoixtiw) J fc(«) dt = 0, i = 1, 2, • • -, (133) 

and 

^oU.-(w) €/(«)} = f f ^o{.r s (w).T < (co)}^(s)^(0 dsdt 

J T J t* 

= f f r (s,t)Ms) MO ds dt (134) 

J 7" J 7* 

= XAy; tj = 1,2, .-., 

where Mercer's theorem is used for the third equality. Then, since 
£,-(&>), * = 1, ■ ■ • , n, are Gaussian variables, (133) and (134) immedi- 
ately give (76) and (77) with m = 0. 

C.2 With respect to Pi 

By substituting ( 129 ) into ( 132 ) , 

&(«) = 2>*.-ifc(«), a - e - (Po,Pi) (135) 

where 

ij,-(w) = f x*(«)p.-(0 <ft, * = 1, 2, • ■ •,* (136) 

which exist a.e. (Po,Pi), and Gaussian variables just as £,-(«), i = 
1,2, • • • , are. Then, the results in C.l imply that 

Ei{ 77,(0.) tu(w)) = n£ni i,j = 1, 2, ••• . (137) 

Define 

m 

€i < " , («) = E ««**(«), ./=!,•••, », (138) 



* Note 77,(0)) here must not be confused with the one in Section 2.3.2. 
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and let Fi (to) be the distribution function of fi (m) («), ••• , £» (m) (o>), 
and let /i (m) (ri , • • • , t„), — °o < tj < <x>, j = 1, • • • , n, be its char- 
acteristic function with respect to P x , i.e., 

/i lra) (n , • • " , r») = E 1 jexp [t g r^ (m) (co)]J . (139) 

Then, according to Levy's continuity theorem,* lim Fi exists if and 

m-*oo 

only if lim/i (m) (t } , • • • , t„) exists for eveiy t } ■■ , — °o < t,- < °o, and 

m-»oo 

continuous at 77 = 0, y = 1, • ■ ■ , n\ and, furthermore, when lim Fi (m) = 
F t exists, its characteristic function /i(ti , • • • , r„) is equal to lim /i <m) 

m-*oo 

(ti , • ■ • , r„) for all tj , — oo < r j < oo, j = 1, • • • , n. Hence, 
it suffices to obtain lim /i ( "°(ti , • • • , r„), namely, the limit of (139) 

m->oo 

asm-> co , and to assure its continuity at the origin. 
By substituting (138) into (139), 

V m) (ri, • • •, Tn ) = E, jexp U g r y g u*fl*(o>)J| 

= £?i <exp X) *W<«0 £ TjU W 

= II exp - ^ Mft ( g ryti4/J 

[« to n n ~1 

-^EZZ TiTjUkUkiUkj . 
^ A-=l ,=1 j=l J 



Note that 



since 



2_/ I WcUkiUkj \ = 22 Wc\ UkiUk 

lc=l h=l 



J | = Z) Mfc < °° , 



(138) 



uwujtf 1 = 1/ <Pk(t)fi(t) dt\\J <pk(t)fy(t) 



=" / *>* 



= 1 . 



! (0 eft 



eft 



\Pi(t) dt \ xfsj (t) dt 



* See Cramer, 18 p. 102. 
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Hence, 

E £ E TiTjUkUkiUkj = E E TiTj 2j V-kUhiUkj. 

k=\ 1=1 i=\ 1=1 J=l fc=l 

Then, putting 

00 

(Qi (n) ).v = £ WfcW*»M*i ; *f i = li ••• i »f 

A=l 

continuity of exponential functions implies 

lim / (m) (r, , • • • , r„) = exp T-± E E (flk w W, , 

TO-»00 L * « =1 J = 1 

which is obviously continuous at t, = 0, i = 1, • • • , n. Note that the 
right-hand side above is the characteristic function of the Gaussian 
distribution function with the density function (76) and (77) with 
m = 1. 

APPENDIX D 

Po — Equivalence between ($> T and (S M 

It is to be proved that, for an arbitrary set A 6 (B T , there exists a 
nonempty set A. € <&„ such that P (AaA) = 0. Note, however, that 
the above statement is equivalent to the following: 

Let JFt C (Br be a class of all sets A € (B r such that A^ 5 T implies 
existence of a nonempty set A £ (&„ with P (AAA) = 0. Then, ffr = 
(Br-* 

The second statement will be proved. 
D.l For every t <E T, 



X t (u) = E &(«)**(<), a.e.(Po). 
k=i 



(139; 



Proof: 

According to the discussion in Section 2.1, (ii), £*(o>), k = 1,2, • • • , 
are equal, a.e. (Po), to the Riemann integrals in quadratic mean cri- 
terion of x t {o))\pk(t) on T. Hence, from the proper orthogonal decomposi- 

* It must be proved first that such an S T is not empty. This will be done in 
Section D.2. 
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tion theorem,* the series of (139) converges in quadratic mean with 
respect to P to £*(«) uniformly on T. Furthermore, &(w), k = 1, 2, • • • , 
are mutually independent Gaussian variables with means zero and vari- 
ances Xa . , k = 1,2, ■ ■ • , with respect to P .f Hence, the series converges, 
a.e. (Po), to a limit for every t 6 T since the series of its variances con- 
verges for every t € T, i.e., 

£ B46?(»)}+f(t) = EW)W) = r (M) < oo, 

from Mercer's theorem. Yet, since both the convergence in quadratic 
mean and the convergence almost everywhere imply the convergence 
in probability measure, this limit must be equal, a.e. (Po), to .r,(w) 
for every t £ T. (Q.E.D.) 

D.2 Let A T £ (Br be denned by 

Ar = {x t .(a) g pi : , i = 1, • • • , «}. (140) 

Then there exists a nonempty set A T 6 &„ such that 

Po(ArAAr) = 0. 

Proof: 

Consider a set defined by 



Ar = ill ZMMU) ^ Pi, i = 1, ••• , n>. (141) 

Clearly, Ar £ (&„ . Define r ( 6 @>t by 

r, = L(a,) = g &(«)*k(o} , t € r. (142) 

Note that (139) implies 

p„(i\) = i, t <e t. (143) 

Then it is self-evident that, for U € T, i = 1, • ■ ■ , n, 

A T = A T f\ ( I\J + A r fl ( U r„ - Y 

Ar = Ar n(n T J+ Ar H ( U T t A , 

where I\. c is the complement of I\ . Note that, from (142), 

* See LoeVe, 10 pp. 478-479. 
t See Appendix C.l. 



(144) 
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a 7 . n (h v t \ = A r n ( n rA (145) 

and, from (143), 

p ° [ Ar n (ja r '- c )] = ° = p ° i^ T n (h r '' e )] ■ (146) 

Hence, upon combination of 144, 145, and (146), 

Po(ArAAr) = 0. (Q.E.D.) 

D.3 ffr = (B T • 

Proo/V 

First, it is easily seen that the class 5 T is a field. Moreover, it will now 
be shown that JFy is a Borel field. Let A, 6 JFr, i = 1, 2, • • • . Then, 
from the definition of 5F T , there exists At € ®„ such that 

Po(A,AA,) =0, i = 1,2, ••■ . (147) 

Define two sequences of null sets il/, and Ni , i = 1, 2, • • • , by 

il/i = A, - A, , A r , = A, - A, . (148) 

Then, 

A, - Ni C A, C A, U ili, , i = 1, 2, • • • . 

Hence, 

U A, - U ATi C U A, ■ C f U A,) U ( U M.) , 
t =i ,=i t=i \»=i / V=i / 

which implies 

U A, - U A,C U Ni, 
i=i i=i i=i 



Thus, 



namely, 



U A, - U A.C U Mi. 

i=l i=l «=-=! 



^S^M]-* 



u a, € g>. 

i=i 
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Furthermore, since 
and also 3> is a field, 

X 

n a,- e 5t . 
t=i 

Hence, *5 T is a Borel field. 

Secondly, note that fF r contains the generating class of 03 T as shown 
by (140) and (35). Hence, 

£F T 3 (Br . 
Yet, from the definition of JF r , 

5 r C(B T . 
Therefore, 

ffr = (B r . (Q.E.D.) 

APPENDIX E 

Equivalence between Two Equations 
E. 1 Preliminary 

Through Mercer's theorem, 

ro(s,<) = £ X t -fc(a)fc(0 , (149) 

uniformly. 

ri(«,0 = Zw(8W/), (150) 



Then, 



Hence, 



/ / ri(s,t)\f/,(s)tj(t) ds dt = 2 UkUkiUkj = a,-,-. 

•t-'t t=l 



n(M) = IIflui(s)W), in the mean. (151)* 

* This is a trivial extension of well-known results in the case of functions of one 
variable. A special case of (151) is found in Courant and Hilbert, 19 pp. 73-74. 
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E.2 Equivalence between Two Equations 
Equation (95) can bo rewritten as 

oo 

E \faiflki = «o - M.-i , j = 1,2, ■■■ , (152) 

A=l 

where i = 1, 2, • • • . Repeating (100), 

[ ( r (s,u)h(u,v)n(v,t) du dv = n(s,t) - r„(s,0 . (153) 

(a) If (h a , h a , ■ • • ) is a solution of ( 152) for each i = 1, 2, • • • , 



wi 



ith 



then a square-integrable function fc(«,t) with 

X 00 

*(«!*)■ L E M<(«)*i(0, in the mean, (155) 

i=l ;=1 

satisfies (153). 

Proof: 

The left-hand side of (15:5) is clearly square-integrable. Hence, it 
has the following expansion: 

oo oo « 

r9(8,u)h(ujB)ri(v t t)dudu^ EEEU^WW, , v 
J r Jr i-i y-i fc-i (156) 

in the mean , 
since, through substitution of (149), (150), and (155), 

fill! r o(8,u)fc(M,»)r 1 («,OdMrfJ^,(s)^(Od»^ 

(157) 
= E A«^«A«A> ; i J = 1, 2, • • • . 

A=l 

Yet, by virtue of (h n ,h i2 , ■■■) being a solution of (152) for each 
i = 1, 2, • • • , the right-hand side of (156) becomes 

E E E \jhnflvin(8)Mt) = E E (a« - MtMMtod) , d«8) 

, = 1 j_l A = l 1=1 >=1 

the right-hand side of which in turn becomes, from (149) and (150), 
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ri(s,0 - r (s,t) 

00 00 

= EZ (a,/ - Xrf*y)to(*)*i(Oi in the mean. (159) 

i 1 ;- 1 

Thus, upon combination of (156), (158), and (159),* 

/ / ro(s,u)h(u,v)n(v,t) dudv = ri(s,t) — r (s,t) . (160) 

(Q.E.D.) 

(b) If h(s,t) is a square - integrable solution of (153), then 
(hn , h i2 , • • • ) satisfies (152) for each i = 1, 2, ■ ■ ■ , where 

h,j = [ f h(s,t)ii(s)h(t) dsdt. (161) 

,'t Jt 

Proof: 

Since h(s,t) is s(|uare-integrable, it has the expansion of (155) where 
hy j hj = 1> 2, • • • , are defined by (161); thus (157) is established. 
Meanwhile, from (149) and (150), 

f [ [n(s,t) - n,(«,0]*i(«)to(0 - cm ~ Mir, 

(4 a 1 2 • • 

Then, combination of (160), (157) and (162) establishes 



(162) 



J\ \ t h ik a kj = an - XJij . (Q.E.D.) 

APPENDIX F 

Alternative Conditions 
Assume 

a it < 1, i = 1,2, ■■■ , (163) 

and the integral equation (100) 

/ / r (s,u)h(u,v)ri(v,t) dudv = n(s,t) — r (s,t) (164) 

Jt Jt 

has a square-integrable solution, f Then, the conditions that 

* Note that, if a sequence of functions converges in the mean to two limits, the 
limits are equal almost everywhere. Furthermore, if the limits are continuous, 
they are equal everywhere. Note also that continuity of the left-hand side of 
(15G) can easily be seen through the use of the Schwartz inequality. 

t Recall from Appendix E that this implies £»-i Sj-i hii* < °°. 
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an > Z'|«ol, (165)* 

and there exists a constant K > 0, independent of ij = 1, 2, • • • , such 
that 

| (OiiM - in I ^ K L }J - g' | a jk \j , (166) 

imply the conditions (98), (99) and (88); namely, for each i = 1, 2, 
s 

E|c«|<l, (167) 

and there exists a constant A\ > such that 

bjii) ^ Ki (l - E I c ik |) , j = 1, 2, • • • , (168) 

and finally 

lim tr [(Q (n) ) -, Qi (B) -/]<•, 

n-*x> 

lim tr [Qo (n) (Qi (n) ) _1 - /] < «>• 



(169) 



Proof: 

First, note that 

a« > 0, z = 1,2, ••• , (170) 

and 

X>„ = E«. (i7i) 

1=1 1=1 

For, from (93) and the fact that /z fe > 0, k = 1, 2, • ■ • , 

da = X /**«*•* > 0, 

A=l 

and, from (131), 

oo CO 00 co oo co 

E «n- = ZE wfcffti* = 2 ma- XI w*,- 2 = E jufc.t 

i=l i=l A=l A=l t=l A=l 



* The prime symbolizes omission of the term j = i in the summation, 
f For justification of interchange of order of summation, see Apostol, 20 pp. 
374-375. 
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Second, through (93) and (94) with (163), (165) and (166), 

M 00 

1 - E I c.v I = 1 ~ E I 8u - a-a | 
i-i >=i 

= 1 - | 1 - an | - E' I a u I 
»-i 

= O,,- — S' I ffl ii I > 0» 
l-l 



and 

6,(«) | (o w A<) - 5, 



1 — E I Cy* | a# — E' I Q-ik | 

fc-1 *=1 

which prove (167) and (168). 

Last, note from the definition of ha " , i,j = 1,2, • • • , 

tr [(Qo ln) )- 1 Qi (B) - /] = E E hi^au = EZW\, 

i=i y=i *=i j=i 

- tr WiQ^r 1 - i] = EEC'Wv = IX- (n) x.-. 

,;=1 j=i i=\ 

Yet, according to the theory of infinite systems of equations, for each 
i - 1, 2, • ■ ■ , 

|W" | £#,, j- 1,2, ..■ .f 
By putting /<", = K, i = 1, 2, • ■ • , 

|W n) | g/v; *j = 1,2, ... . 
Then, 

ZZlWVi ^ «EElay| < •, 

i=i y—i i=i i-=i 

CO OO 

ZlW°x, I g«E^< °°. 



since 



* Recall: 

(Qo<»>)-W° - / - [(Go 00 )" 1 - (<?i ( " , )- , ]Qi ( " , I 

W° = /[(<?o (n >)- 1 - (Qi'"')-'].,- ; iJ = 1, ■■■ ,n, 
\0; * ,j = n + 1, n + 2, ••■ . 

t See Kantorovich and Krylov, 16 pp. 2G-27. 
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££\av\ -£(a,i + Z'|a I y| > ) < 2 £ a,-, = 2 £ M . < oo. 

i=i >=i i=i \ >=i / *=i i=i 

Hence, from (97), 

lim | tr {(Q M r l Q^ - I] \ = lim 



i=i j=i 



1=1 >=1 



< 



co cc V S / og co V 

E E *»•) (E E «./) 

i=i >=i / \i=i y=i / 



< oo 



lim |tr[Qb w (Qi ) - /] | = Urn 

n-»oo n-»oo 



i=l 

i=i 

g (i>-')'(£ x <)' 



< oo. 



(Q.E.D.) 
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